Agreed it's unlikely, but wouldn't it be possible to achieve 100% locality for both primary and secondary replicas if your number of replicas is less than or equal to your hdfs replication factor?

Theoretically that's possible. I say it's unlikely because something would need to ensure that all the block replicas for a secondary region end up co-located with the region, and nothing I'm aware of would do that. Compactions won't help. Internally at hubspot we have the locality healer, but today even that doesn't know about secondary regions, although I plan on that eventually.

The balancer is aware of locality — I wonder to what extent LocalityBasedCandidateGenerator could maintain secondary replica locality. I haven't looked at the implementation with replicas in mind, but it might already balance secondary replicas based on locality to some extent

True. I would still argue that that isn't likely to be very helpful. The region balancer needs to put a region on exactly one server, but in the absence of some helpful force, it's unlikely that all the blocks for any given region exist on a single DataNode. The region balancer can only do as well as the random block distribution allows. Before we had the locality healer at hubspot, we experienced this difficultly all the time.

Yeah good point